Repository navigation
Persist bounded lead facts with checkpoint replay and stall incidents (#629) - #684
Conversation
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f765d3b06a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
| snapshot.as_of_seq = event.seq; | ||
| snapshot.at_ms = event.at_ms; | ||
| snapshot.to_json().map_err(|error| error.to_string())?; |
There was a problem hiding this comment.
Trim retained history before rejecting the snapshot byte limit
When the situation reaches the 64 KiB byte limit before reaching 128 rows, serialization rejects the observation without attempting to evict retained landed/incident history. For example, 95 valid asks with 512-character questions plus 16 incidents with bounded identifiers exceed 64 KiB, while those same active asks without incident history occupy approximately 56 KiB. Because append must succeed before the runner's next action, disposable history can unnecessarily stop progress. Apply history eviction for the byte limit as well as the row limit, setting history_truncated, and reject only when the active projection itself cannot fit.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 65f463c (failing-first regression f0da0e0). Projection now discards only landed/incident history until both row and serialized-byte limits fit, marking history_truncated; active rows are never removed. The real-store regression reproduces the original failure with 95 long asks plus 16 incidents, then proves asks survive, reopen agrees, and active-only overflow refuses without advancing the sequence or changing the prior snapshot. All 18 focused storage/stall tests pass; full workspace verification is running.
| io( | ||
| fs::DirBuilder::new().mode(0o700).create(path), | ||
| "cannot create facts root", | ||
| )?; |
There was a problem hiding this comment.
Sync the parent directory after creating the facts root
When initialization creates the facts directory, its entry in the containing repository directory is never synced. Subsequent Root::sync calls sync only the new facts directory, which does not guarantee durability of its entry in its parent. A power loss shortly after successful initialization and acknowledged appends can therefore lose the entire store despite its files having been synced. Sync the containing directory after creating the root before reporting successful initialization.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 65f463c (failing-first regression f0da0e0). Explicit initialization syncs the containing directory before any successful initialization, including retries where mkdir already happened. The existing child-directory sync remains on file publication. A real permission regression makes the parent writable/searchable but unreadable: the old implementation initialized successfully, while the fixed implementation refuses the failed parent-directory open/sync. It runs in a subprocess and drops root privileges when necessary; no test skip. All 18 focused storage/stall tests pass; full workspace verification is running.
|
First-pass review by oxa → oxc → Codex findings
🤖 Generated with Claude Code (oxa review lane) |
|
gatekeeper: Independent read of f765d3b. Findings: no P1–P3 findings in this library scope. Checked the entire diff, shared fact codecs and validation, journal/snapshot recovery ordering, rotation boundaries, checkpoint deduplication, bounded source/active/history tables, cross-handle lock acquisition, and private file operations. Pending state precedes paid-for durability claims; the journal is synced before the source cursor commits, and lagging projections recover before another append. Tests cover torn writes, post-append and post-commit recovery, interrupted rotation, lost acknowledgements, concurrent emitters, retention gaps, source/active capacity, poisoned leaves and explicit unblocking. Worker completion remains distinct from acceptance/publication/merge. Stall detection validates invocation identity and returns an observation without killing or polling. Fixed storage bounds and no background runtime are proportionate; the reported store-use CPU cost is visible rather than hidden behind idle CLI measurements. Named gate is green; no builds run. Runner adoption and real CLI boundary proof remain follow-up work. Touches oxa-work: the orchestrator must still perform its own independent read after this comment. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 65f463c876
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } else if journal.events.last() != Some(&event) || journal.torn_at.is_some() { | ||
| return Err("pending fact disagrees with the journal".into()); | ||
| } | ||
| state.committed(&event)?; | ||
| self.root.atomic(STATE, &state.bytes()?)?; |
There was a problem hiding this comment.
Sync an already-written pending event before committing recovery
If the writer exits after writing the complete journal line but before Root::append finishes sync_all (or that sync returns an error), recovery can see the matching line in the page cache and take this branch without syncing it. Recovery then durably clears pending and acknowledges the observation, although the journal data is not guaranteed durable; syncing the metadata replacement and directory does not sync the journal's contents. A subsequent power loss can leave committed metadata ahead of the journal, making the store fail to reopen. Sync the existing matching journal line before committing the recovered state, just as the replay-and-append path does.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed and fixed in 667f15a (committed red regression d325ab2). The matching complete-line recovery branch now opens and syncs the actual retained journal before committing the source cursor/clearing pending. A permission regression failed before/pass after; all 19 focused tests pass. A separate native strace fault probe proves the durability boundary: the old head acknowledged with no journal fsync; the fix receives an injected EIO from that journal's fsync, leaves the complete pending metadata byte-for-byte unchanged, and then recovers exactly one event on an uninjected retry. This was also proved when the matching event is in the previous journal with an empty current journal. Full workspace/current-head gate are running; PR stays draft meanwhile.
|
First-pass review by oxa → oxc → Codex findings
🤖 Generated with Claude Code (oxa review lane) |
|
gatekeeper: Independent reread of 65f463c. Findings: no P1–P3 findings at this head. Rechecked the complete effective diff against the prior reviewed head and reread the affected projection/initialization paths. Disposable history now shrinks for the encoded-byte cap as well as row count; active-only overflow refuses before reserving a sequence, with unchanged snapshot asserted. Initialization syncs the containing directory even on retry, and sync errors propagate. The new tests exercise actual capacity and permission failures. The remainder of the full diff is unchanged from my earlier journal/recovery, concurrency, private-file, stall and resource review. Named gate is green. No builds or credential files read. Touches oxa-work: the orchestrator must still perform its own independent read after this comment. |
|
gatekeeper: Independent reread of 667f15a. Findings: no P1–P3 findings at this head. Rechecked the full diff against previously reviewed 65f463c and reread pending recovery ordering. A complete visible pending line is now synced again before committing metadata/acknowledging it, using the current or retained journal as appropriate. The real permission-failure regression proves recovery leaves state/projection unchanged when that sync cannot be established and succeeds after access is restored. The prior bounds, history trimming, private-file, serialization and no-background-runtime assessment remains applicable. Named gate is green. No builds or credential files read. Touches oxa-work: the orchestrator must still perform its own independent read after this comment. |
|
No P1–P3 findings at this head. Read the complete facts implementation in prior passes and every change since: the facts/storage code remains unchanged from my last MERGE-OK. The added synthetic gateway fixture correction uses a process-local atomic sequence plus exclusive mkdir/retry, eliminating timestamp collisions without sharing fixture directories or weakening isolation. Journal durability/recovery, bounded history and pending checkpoint replay retain the previously reviewed behavior. No builds or credential reads; named gate green. oxa-work still requires the orchestrator own independent read after mine. |
# Conflicts: # oxa/oxa-work/src/lib.rs
|
No P1–P3 findings. Read the current patch relative to the previously reviewed facts-storage patch: storage/situation logic is unchanged; fixture changes add the required preparing step to task-start observations, with other differences from the updated base. Checked bounded persistence/replay, checkpoint deduplication, trimming/gap reporting and durable append/directory-sync controls. Local acceptance, external acceptance and actual merge evidence remain separate. No builds/tests run locally. The orchestrator still needs its own oxa-work read. |
|
No P1–P3 findings. Read the complete resolution delta against ab952f4. Facts persistence/replay, trimming, checkpoint semantics, situation projection and tests are unchanged; protocol context/lockfile compose the new base and the redundant fixture-root modification drops out. Checked that TaskStarted/TaskTransition and explicit merge evidence retain the previously reviewed semantics. No builds/tests run locally. The orchestrator still needs its own oxa-work read; no approval is implied. |
|
O411 primary disposition handoff; existing sole writer retained. SPLIT:1704 own lines. Existing worker-contract/facts writer remains sole writer; lead will only consume the public API. Extract existing protected facts_io212-line source with focused security tests, then projection/incidents and journal/replay slices <=400 each, maintaining atomic checkpoint/cursor semantics and existing public Facts::situation contract. Proto removals inspected: Running/Incident record declarations are extended, not dropped. Do not copy facts modules into lead or split away required crash invariants. No branch edit, closure, archive or repeat full review by this disposition owner. Release #763 and security O413 remain first. |
|
O417/O418 supersedes the size-only SPLIT request in my previous O411 handoff. Stop forced size splits; retain current sole writer and any split work already made. Existing reported checks are green; use existing review evidence for a gate re-verdict and normal ready/required CI where still draft. No new full third lane review. Grandfathered size alone is not a blocker. Real findings, parent dependencies, owner-read and authority protections remain; READY here is author readiness, not merge authorization. No branch change or merge by this disposition owner. |
|
@codex security review |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
|
READY FOR OWNER READ — exact head 4cf042b. Current GitHub state: ready, MERGEABLE, no failed or pending checks. Codex code summary 6915322560 reports no findings/unresolved findings at this head; canonical security receipt 6949516423 explicitly reports no security issues at 4cf042b. No repeat review or head refresh is needed. Narrow read scope: protected facts-root placement outside worker-writable binds, no-follow/ownership checks, journal reservation/recovery/ack ordering and durable directory sync. This is library-only: no CLI activation/store/thread on the normal path; preserve default-off orchestrator activation. Read #684 before dependent #759. No merge performed. |
The lead needs current facts without reading private runner state or interpreting logs. This adds the L1 events, situation and incidents modules in oxa-work, consuming #669's shared types. The runner retains task state, transition emission, checkpoint adapters and CLI/action wiring.
Tier T3 for protected durable files, rotation and recovery. The exact manifest hunk proposes only existing workspace oxa-proto/libc edges and their lock edges (shared types and no-follow I/O); no dependency version is added. Manifest-owner review is requested as part of this PR.
Includes the requested TaskTransition/TaskStep/StepReason extension and explicit Unblocked observations. API, recovery and adapter contract.
Validation on f8753be, after a normal merge of main 30a2d21:
On the unchanged storage implementation before this main refresh, a native public-API probe (isolated synthetic files, stripped Rust 1.99.1) measured: 4,000 appends plus situation/page/incident reads; one thread throughout. RSS was 1,188 KiB before initialization, 1,388 after, 1,936 after 4,000/read; observed peak 2,480 KiB, cumulative CPU 31.13s. Journals ended at 118,721 / 261,958 bytes, metadata at 290 and situation at 238; sequence=4,000, one emitter, no pending observation, explicit retention gap. The normal append path avoids duplicate complete replay. This is synthetic filesystem work, not model/provider performance.
Private composition with runner #69211dc511 also passed real task/check/save through the public callback: lost-ack recovery, nine contiguous facts, one worker completion, saved session and a finished retry with no new work. Independent proof now drives immutable runner preview 57a5c0c through literal work situation (text/JSON) and work events --after N while a real child waits on a loopback model request. It shows the live task/host/branch/worktree/saved session at cursor4, clears running on completion at cursor9 without claiming acceptance/landing, and a cached retry makes zero requests/events. A planning refusal becomes an explicit Unblocked after the missing Git ref is restored with unchanged task identity. The same immutable preview passes all 3,608 workspace tests, inventory/report guard with zero runtime skips, doctests, strict workspace/all-target clippy, fmt, unchanged scans and document links. Earlier crash fixtures now fail at real checkpoint callbacks rather than rewinding finished state. No task-state/history workaround or invariant relaxation was applied. This is preview integration proof; the runner's eventual main-based PR needs its own current-head checks.
Default CLI footprint (module not wired into CLI yet): same build path/toolchain, stripped release versus main 32297f7; 20 interleaved ABBA saved-session RPC/mock samples with confirmed active streaming.
The default path creates no facts store or new thread. Anonymous medians are identical; the +6 KiB idle / +70 KiB streaming total-RSS medians come from the file-backed working set and link layout, within overlapping ranges. Binary size differences are not a claimed optimization. Interleaved timings are noisy (including negative paired differences), so no speed claim is made. Actual store-use cost is measured separately above.
This is L1's library layer, not all of #629. Runner adoption and real work CLI proofs remain in the active goal; L2 standing follows that boundary. The author does not queue or merge.